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1. INTRODUCTION 

Almost all education institutions in Malaysia use lecturers’ evaluation to review and rate lecturers’ 
teaching performance and effectiveness in the classroom. The findings from these evaluations are used to 
provide feedback to lecturers and guide their professional development. It cannot be denied that lecturers or 
academicians are the backbones of education institution, whose, their service will determine the institution 
success and reputation. Dorasamy and Balkaran [1] said that the evaluation not only to rate the teaching 
quality of the lecturers but also important in accessing the programme effectiveness. Yeoh et. al [2] who had 
done a research at private university confirmed that lecturer qualities are the most important indicator in 
determining lecturer performance which can improve students’ satisfaction and the related university image. 
According to Seldin, Miller, and Seldin [3], students are considered as important stakeholders of the higher 
education institution system because they directly involved with the process of receiving knowledge and 
observing the way of their lecturers teaching in the classroom. Sok-Foon et. al. [4] in their study said that the 
lecturers’ evaluation is a good indicator to be used in enhancing lecturers’ development and students learning 
performance. Even though Machingambi and Wadesango [5] recommended that lecturers’ evaluation of 
teaching should be made mandatory, they also stressed that it must be integrated with other multidimensional 
evaluation methods so that the validity and reliability of the evaluation of teaching effectiveness can be 
improved. This point of view is also agreed by Iyamu and Aduwa-Oglebaen [6], Archibong and Nja [7] and 
they thought that it was improper to use the lecturers’ evaluation for promotion or salary increment purposes. 
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Universiti Kuala Lumpur, as a technical university, with a tagline of ‘Where Knowledge is 
Applied’, has also adopted this good practice. Lecturers’ evaluation has become one of the main criteria that 
is looked at for the annual key performance index (KPI) and promotion appraisal. As of now, UniKL has 
been using the relevant elements in assessing evaluation of lecturers’ teaching performance from the 
students’ perspective. The five main elements are lecturer’s commitment, teaching delivery, lecturer’s 
interaction, assessment and course content and material. Several other supporting points that are also being 
evaluated are punctuality, lecturers’ knowledge and teaching approach. All the criteria used in lecturers’ 
evaluation have the same weightage. Therefore, students’ feedback might not fully reflect the actual teaching 
efficiency of the lecturers. An upward reweighting of the feedback scores of each criterion in lecturers’ 
evaluation to ensure more parity in the teacher evaluation process should be proposed, so that the most 
important element will be given the highest weightage and so on. 

AHP is proposed to be used as a tool to improve lecturers’ score calculation by determining the new 
weightage for each criterion in lecturers’ evaluation. The ambiguity of concepts that are associated with 
human being’s judgments will be used to prioritize the criteria [8]. AHP is very flexible and is broadly 
employed in various fields to optimise distribution network, technology transfer, integrated manufacturing 
and performance appraisal [9, 10, 11]. 


2. LITERATURE REVIEW 

Teaching is, forever will be, one of the major duty of all universities. Thus, the quality of teaching 
should always be one of the fundamental objectives for the universities. Therefore, there is a need to assess 
teaching performance. The purposes of evaluating teaching performance are to build up lecturers’ 
professionalisme, to support self-improvement and to look at suitable continuous quality improvement [9]. 
Assessing teaching performance is not a simple task, as it engage with human decision-making, which is 
loose, dubious and very subjective. Thus, using the scientific method to assess teaching performance 
exhaustively and adequately plays a vital role in deciding the quality of teaching performance. 

Eduardo and Bruno [13] agreed to the idea that students’ evaluation is used by some institutions as 
part of instruments for an academician to be promoted but they believed the questions or items asked in the 
evaluation form could be distorted in some undesirable ways especially by the variables that are beyond the 
lecturer’s control. That is why, there is a need to look into and modify the items in students’ evaluation form 
to avoid an academician being promoted or not, unfairly. 

The adequacy of the evaluation process depends mostly on the proper design and assessment of the 
evaluation. Successful evaluation mechanism demands consideration in recognizing competencies of 
lecturers as well as establishing evaluation criteria specific to a different element of skills [14]. Lecturers’ 
evaluation is very important for lecturers to internalize the students’ feedback and make appropriate changes 
to their teaching styles and it is also important in strengthening the management of the university. The 
university can only give outstanding service to the community if it focuses on continuous quality 
improvement [12]. 

Besides teaching, lecturers are also obligated to meet students’ expectation, maintain or improve the 
ratings of the university and to ensure that the graduates are well received in the job market [15]. Yin, Wang, 
and Han [16] investigation on the Chinese universities students assess on their lecturers’ teaching revealed 
that favourite lecturers were described as knowledgeable about the educational curriculum, engage with the 
students, teaching beyond textbooks and examinations and adopting more students-centred teaching 
approaches. 

Many studies have been done by researchers to identify the most suitable and appropriate elements 
to be included in lecturers’ evaluation for the purpose of determining the lecturers’ teaching competencies 
and professionalism. It can be a crucial reference to enhance the teaching quality offered. According to 
Vevere and Kozlinskis [17], the four key factors that are needed in lecturers’ evaluation are knowledge 
transfer, knowledge evaluation, lecturers’ personal traits as well as lecturers’ accessibility. They also stressed 
that the relationship between lecturers and students is the most important item in analysing the lecturers’ 
performance because the attitude of lecturers has a high impact towards the students’ ability in their learning 
process. 

In 2015, a study which was done by Phillips, Balan and Manko [18], suggested that there were five 
crucial components in measuring the teaching performance. The components are classified as lecturers’ 
planning and preparation, an environment of the study place, instructional delivery, professionalism and 
collaboration and partnership. However, only the first four components were meant to be rated by students as 
the last component has no relation with teaching and learning. 

In Malaysia, other higher institutional educations also opted to execute the teaching evaluation aside 
from Universiti Kuala Lumpur. Universiti Teknologi Malaysia for example, has implemented Lecturer 
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Performance Assessment (ePPP) which look into four main parts; i) Planning and Preparation ii) Delivery 
Methods iii) Assessment and iv) Students-Lecturers Relation. The ePPP also allowed the respondents to give 
a view in the comment section for detailed elaborations [19]. In the same research paper, Samian and Noor 
also emphasized that the students’ achievements had a high positive correlation with two criteria that are the 
delivery skills and the relation of lecturers and students. They, at once, disagreed with the claim that the 
lecturers’ evaluations are invalid when it comes to be a benchmarking of teaching effectiveness, as stated in 
some studies. The traditional method of teaching is still powerful and crucial in getting beneficial information 
on lecturers’ teaching performance. 

As the lecturers’ evaluation is worldwide used and accepted, a few researchers have concentrated on 
assessing teaching performance in universities and several methods have been used in enhancing the details 
in lecturers’ evaluation as well as the calculation of the data obtained. Previous studies mainly focused on the 
details of students’ evaluation with minority examined the quantitative method on getting the information 
based on the students’ evaluation [20]. Since the students’ evaluation consist of many different criteria, the 
suitable method of accessing the correct information from data obtained is crucial. 

AHP, a well-known tool in aiding in decision-making process when dealing with multiple criteria. 
This method is considered as a good method that can solve complicated decision matters with easily 
accessible input data [21]. Recently, Yousif and Shaout [22] combined AHP and TOPSIS method to evaluate 
teaching quality. TOPSIS was introduced to overcome the pairwise comparison in AHP due to uncertainty in 
the decision maker’s judgement. The combined method has been a good application of AHP theory in 
evaluating teaching performance. Chen, Hsieh and Do [12] also proposed an approach of teaching 
performance evaluation with fuzzy AHP. This study focuses on improving lecturers’ evaluation criteria or 
items, applicable in Universiti Kuala Lumpur by using the AHP. 


3. RESEARCH METHOD 
3.1. Research Design 

Data was collected by distributing a questionnaire to a few decision makers who are responsible in 
designing and implementation of this evaluation. Among the respondents are Deputy Director, 
Senior Manager and Coordinator of UniKL Centre of Instructional Technology and Curriculum (CITC). 
The input from Deputy Dean Academic and Technology and Heads of Section of UniKL Malaysian Institute 
of Information Technology (UniKL MIIT) also have been considered to streghthen the outcome of study. 


3.2. Criteria for Lecturers’ Evaluation 

The lecturers’ evaluation is carried out once in every semester by all institutes of UniKL through the 
online system. Five criteria are used as a domain of evaluation and is illustrated in Figure 1. Further 
explanation about the criteria used in lecturers’ evaluation are shown in Table 1. 
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Figure |. Lecturers’ Evaluation Criteria 
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Table 1. Explanation about Lecturers’ Evaluation Criteria 








Criteria Explanation 
Lecturer’s Lecturer has good time management, always on time in each task given, very concern about 
commitment students’ academic achievement, arranges immediately class replacement and provide constructive 


feedback to students on each assessment. 

Teaching delivery Lecturer uses English as a medium of interaction while teaching, well-prepared, knowledgeable, 
enthusiastic, explains well the subject matters through real life situations and uses variety of 
teaching methods in conducting lecture. 


Lecturer’s interaction — Lecturer constantly encourages students’ participation in class, provide sufficient time for 
consultation, approachable and ready to entertain students’ questions regarding to subject taught. 


Assessment Lecturer assesses students’ cognitive and soft skill based on the learning outcomes of the course. 
Lecturer also always firm and fair in evaluating students’ assessments. 

Course content and Content and teaching materials delivered by lecturer are neatly prepared and easily understood and 

course material. meet the learning outcomes of the course. 





3.3. Analysis Method 

AHP method by Saaty [8] has been used in this study to determine the priorities of importance 
(weights) to the main criteria of the lecturers’ evaluation. This method is an effective approach in decision 
making for selecting the best alternative to identify the overall score of lecturers’ evaluation. 

AHP provides a structured framework for setting priorities by using pairwise comparisons. 
Comparing the process of criteria for their relative importance is necessary to be been done as shown in 
Figure 2. The relative importance using | to 9 scale of criteria for every paired criteria is illustrated 
in Table 2. 


Table 2. Saaty's Scale of Relative Importance 








Importance Intensity Definition 
1 Equal importance 
3 Moderate importance of one ever another 
=) Strong importance of one over another 
a Very strong importance of one ever another 
9 Extreme importance of one over another 
2,4, 6,8 Intermediate values 
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Figure 2. Pair wise Comparison Matrix 


The implementation of AHP continued by the normalization of pair wise comparison. 
The normalisation of comparison matrix was computed by dividing each entry of the column with the 
summision of entries for every column of comparison matrix as shown in Figure 3 and Figure 4. In the 
following process, the mean value of the numbers on each row of the normalized comparisons matrix is 
computed. The calculated mean values represent the relative weight of decision elements with matrix rows as 
shown in Figure 5. Multiplication Matrix C and Weight as shown in Figure 6. 
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Figure 5. Relative Weight Figure 6. Multiplication Matrix C and Weight 


i) 


ii) 


iii) 


iv) 


Consistency Ratio (CR) will be used to determine the consistency and reliability level of the result. 
Pairwise comparison may look simple, but when the number of comparisons arises, the level of confidence 
on the consistency of comparison is not simply accepted and it requires the CR. According to Dyer and 
Forman [23], if the reading of CR is less than 0.1, the consistency of comparisons will be acceptable and it 
can be concluded that all the judgements are consistent and valid, otherwise the criteria comparisons should 
be reconsidered. The following steps are needed to calculate the value of CR: 


Multiply matrix C with weight, “ as shown in Figure 6. 


Cir C12 3 Cg 15 |] Wy 
Cor C22 €23 C24 Ca5 || Wo 
CW =] C3, C32 C33, C34 C35 |] W3 
Car Cg2 C43 C4q Cys || Wa 


C5, C52 C53 C54 C55 || Ws 


q _ 1entry-iin Cw 
Compute m5 4 entry-iinw 


Compute the consistency index, 4 


(5) 


(6) 


(7) 


The appropiate random index value is taken from Saaty’s CR random number (RI) index as shown 


in Table 3. 


Table 3. Saaty's CR Random Number Index 











Size of Matrix 2 3 4 5 6 7 8 9 10 
RI 0 052 0.89 1.12 1.26 1.36 1.41 1.46 1.49 
cI 
CR =— 


Compute the consistency ratio, CR by comparing the CI with RI, ~ RI 


(8) 





Improving lecturers’ evaluation score by using analytic hierarchy process (AHP)... (Nurashikin Saaludin) 


396 o ISSN: 2502-4752 


4. RESULT AND DISCUSSION 

UniKL students are required to complete and submit the lecturers’ evaluation form, for every 
lecturer who is teaching them for that particular semester. One of the reasons for implementing the evaluation 
is to measure the teaching quality among UniKL lecturers. In measuring the lecturers’ teaching quality, AHP 
method was used to determine the weightage of each criterion in lecturers’ evaluation based on priority. 
Table 4 below shows the result of analysis. 


Table 4. Comparison Matrices and Weightage for Each of Lecturers’ Evaluation Criteria 
Consistency Ratio = 0.0906 








Criteria A B Cc D E Weightage 
A 1 O.11111 0.25 0.14286 0.125 0.03075 
(3.075%) 

B 9 1 9 3 3 0.46471 
(46.471%) 

Cc 4 O.11111 1 0.125 0.125 0.05648 
(5.648%) 

D 7 0.33333 8 1 1 0.22058 
(22.058%) 

E 8 0.33333 8 1 1 0.22748 
(22.748%) 





Indicator: 

A - Lecturer’s commitment 

B - Teaching delivery 

C - Lecturer’s interaction 

D - Assessment 

E - Course content and course material. 

The finding shows that teaching delivery carries the highest weightage if compared to other criteria. 
To produce quality teaching, a lecturer needs to be knowledgable, enthusiastic and creative in various 
teaching methods. The above criteria formed the largest weightage which is 46% of overall lecturers’ 
evaluation. Meanwhile, assessment and course content and course material criteria is the second and third 
largest criteria with 22.7% & 22.05% weightage respectively. The ranking of these three criteria is agreed by 
Hubackova [24], Chen, Hsieh and Do [12]. According to Hubackova, students rank teacher’s professional 
knowledge in the first place, the contents of material learnt, second and teacher’s pedagogical abilities, third. 
Whereas, Chen, Hsieh and Do opined that well prepared and using a variety of approaches in teaching 
delivery are the most important criteria in lecturers’ performance. The remaining criteria namely lecturer’s 
interaction and lecturer’s commitment formed the balance of 8.7% weightage in lecturer evaluation. 

The overall score for the lecturer is derived by multiplying the weightage of respective criteria and 
the rating given by students to lecturers. The example of the calculation is shown in Table 5. The overall 
score from Table 5 reflects the teaching efficiency of the lecturer since the weightage of criteria is considered 
in calculation as compared to the current method that only use student rating average as an overall score as 
shown in Table 6. 


Table 5. Overall Score Calculation using Weightage from AHP 








Criteria Student’s rating Weightage Score Overall Score 
Lecturer’s commitment 4.60 0.03075 0.14145 4.093 
Teaching delivery 3.55 0.46471 1.649721 

Lecturer’s interaction 4.54 0.05648 0.256419 

Assessment 4.56 0.22058 1.005845 

Course content and 4.57 0.22748 1.039584 


course material. 





Table 6. Overall Score Calculation using Average Student’s Rating Average 








Criteria Student’s rating Overall score 
Lecturer’s commitment 4.60 4.364 
Teaching delivery 3.55 

Lecturer’s interaction 4.54 

Assessment 4.56 

Course content and 4.57 


course material 
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The overall score from Table 5 reflects the teaching efficiency of the lecturer since the weightage of 
criteria is considered in the calculation as compared to the current method that only use student rating 
average as an overall score as shown in Table 6. The proposed method can be used as an indicator of the top 
management to evaluate lecturers from the perspective of students’ point of view. 

The CR is 0.0906. The ratio is less than 0.1 as fixed by Saaty [8]. Hence, the judgment given by 
respondents were considered as consistent, the weightages produced are valid and aligned with other several 
studies Salgado et al. [25, 26]. 


5. CONCLUSION 

AHP has been proven and accepted as an effective and efficient method in decision making 
worldwide. This method incorporates qualitative and quantitative factors that can be used as a tool in 
determining the weightage of criteria for lecturers’ evaluation. The correct weightage for each criterion is 
very important because it will reflect the ranking or priority of the elements being used in measuring the 
teaching efficiency of the lecturers. From the score, lecturers will know the criteria that they should focus 
more and thus, improve accordingly. It will also enable them to disseminate knowledge and skills for the 
students’ benefits. The calculation of inconsistency ratio is a very important feature to ensure the decision 
made cannot be disputed in the outcome. In order for UniKL to drive towards education revolution 4.0, 
lecturers are the backbone who should start the transformation in educating and preparing students for self- 
learning skills, able to think critically, creative and innovative especially to prepare them for the future 
workforce. 
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